Image Loss

1. perceptual loss [1]: two images have similar semantic information

   $$\frac{1}{C_j H_j W_j}||\phi_j(\hat{x})-\phi_j(x)||^2$$

2. style loss [2]: two images have similar channel correlation; related to bilinear pooling [6]

   $$||G_j^{\phi}(\hat{x})-G_j^{\phi}(x)||_F^2$$

   with $G_j^{\phi}(x)_{c,c'}=\frac{1}{C_j H_j W_j}\sum_{h=1}^{H_j}\sum_{w=1}^{W_j}\phi_j(x)_{h,w,c}\phi_j(x)_{h,w,c'}$

3. pairwise mean squared error (PMSE) [3] [4]: scale-invariant mean squared error (in log space)

   $$\frac{1}{n}\sum_i d_i^2 - \frac{1}{n^2}(\sum_i d_i)^2$$

4. total variation (TV) loss [1]: smoothness

   $$\sum_{(i,j)} ||x_{i,j+1}-x_{i,j}||_1 +||x_{i+1,j}-x_{i,j}||_1$$

5. alignment loss [5]: two images have similar spatial correlation, complementary to style loss

   $$||F_j^{\phi}(\hat{x})-F_j^{\phi}(x)||_F^2$$

   with $F_j^{\phi}(x)_{d,d'}=\frac{1}{C_j H_j W_j}\sum_{c=1}^{C}\phi_j(x)_{d,c}\phi_j(x)_{d',c}$

Reference

[1] Johnson, Justin, Alexandre Alahi, and Li Fei-Fei. “Perceptual losses for real-time style transfer and super-resolution.” ECCV, 2016.

[2] Gatys, Leon, Alexander S. Ecker, and Matthias Bethge. “Texture synthesis using convolutional neural networks.” NIPS, 2015.

[3] Eigen, David, Christian Puhrsch, and Rob Fergus. “Depth map prediction from a single image using a multi-scale deep network.” NIPS, 2014.

[4] Bousmalis, Konstantinos, et al. “Unsupervised pixel-level domain adaptation with generative adversarial networks.” CVPR, 2017.

[5] Abavisani, Mahdi, Hamid Reza Vaezi Joze, and Vishal M. Patel. “Improving the performance of unimodal dynamic hand-gesture recognition with multimodal training.” CVPR, 2019.

[6] Lin, Tsung-Yu, Aruni RoyChowdhury, and Subhransu Maji. “Bilinear cnn models for fine-grained visual recognition.” ICCV, 2015.